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Abstract 

The Yale Child Study Center’s Impact Analysis and Strategies Group conducted a three- 
year, independent evaluation of the Institute for Student Achievement (ISA, a multi-site, 
school-based intervention program for students placed at risk of failure). The evaluation 
design triangulated (I) student outcomes, (2) the level and quality of the students’ youth 
development, and (3) the quality of program implementation at the sites. The research 
included direct observations of academic enrichment and counseling sessions; 
retroactive analysis of achievement data; cohort analysis of the class of 2000; a student 
learning and youth development survey; a staff survey; and extensive interviews. The 
quality of implementation varied. The findings show that, when implemented well, there 
is a significant positive “ISA effect ’’ on student outcomes. 

Introduction 

Since the early years of this century, the relationship between youth development and 
student learning has been a topic of heated debate: Is youth development independent of 
learning? Is learning actually development? Are learning and development mutually 
dependent and interactive processes? Or does learning “awaken” development? 
(Vygotsky, 1978). The Institute for Student Achievement (ISA) operates on a simple but 
powerful principle: Focus on both student learning and youth development 

simultaneously. 

Our Impact Analysis and Strategies Group at the Yale Child Study Center is 
currently in its third year of studying the impact of ISA’s school-based academic and 
counseling intervention on the lifepaths of youth. ISA provides school-based academic 
and counseling support services to 2,267 middle and high school students who have been 
identified as being at risk of not making a healthy transition to adulthood and/or being at 
risk of academic failure. ISA student groups are microcosms of the neighborhoods in 
which their schools are located. As such, they present a complex challenge to the adults 
who seek to assist them. Other intervention programs may learn from the challenges and 
triumphs of ISA in promoting the learning and development of the students. 



Problem Analysis 

The Institute for Student Achievement (ISA) was founded in 1990 by Gerard and Lilo 
Leeds, cofounders of CMP Media Inc. ISA is a not-for-profit public/private venture in 
Manhasset, Long Island. Its self-described mission is “to improve the quality of education 
for children and youth at risk so that they can succeed in our society.” Gerard Leeds has 
a persuasive argument to justify the expenditure necessary to attempt this: Provide 
integrated school-based academic and counseling support services to students identified 
as being at risk of dropping out of school, at a cost of $3,000 to $5,000 per student 
annually — approximately $20,000 for a 4-6-year program — and eventually “society 
gets its money back every two years for the working life of each person.” In his (1998) 
words, “When [ISA students] graduate from college, they’ll be making $30,000 to 
$40,000 a year. They’ll average paying $10,000 a year in taxes. When they graduate from 
college, in two years they’ll have paid society back in taxes” (p. 25). Furthermore, society 
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saves by “avoiding the future costs of unemployment, health and welfare support and the 
costs of the criminal justice system” (ibid.). 

ISA is the organization charged with implementing COMET and STAR in 
schools. In April 2000, Dr. Gerry House was appointed President and CEO of ISA. ISA 
is codirected by Patrick G. Halpin and Lavinia T. Dickerson, who designed ISA’s 
COMET/STAR program. COMET (Children of Many Educational Talents) is designed 
to meet the academic and personal developmental needs of seventh and eighth grade 
students. STAR (Success Through Academic Readiness) is the continuation of the 
program in high school. 

Support services are provided by a team generally consisting of a program 
director, academic coordinators, counselors, outreach coordinators, and a college and 
career coordinator (the number of staff and staff roles are determined by the age and 
number of students). ISA students participate to varying degrees in academic enrichment 
and counseling before, during, and after school. COMET/STAR has developed a well- 
formulated and sustained college and career planning curriculum, before- and after- 
school programs that are closely linked to daily classroom experiences, and an extended- 
year component that connects students to learning beyond the regular academic school 
year. The case conference process used by COMET/STAR is consistent with best 
practices in promoting youth development among children placed at risk. 

ISA has traditionally been a forward-looking organization with strategic plans that 
emphasized expansion. During the 1990—1991 school year, ISA served 25 students. By 
1994-1995, ISA served 240 students. During the 2000-2001 school year, ISA provided 
services to 2,267 students in the following school districts and areas: 

❖ Long Island, New York 

• Roosevelt Union Free School District, Long Island 

• Hempstead Union Free School District, Long Island 

• Wyandanch Union Free School District, Long Island 

❖ New York City 

• Community School District 13, Brooklyn 

• Metropolitan Corporate Academy High School 

• Park East High School 

❖ Westchester County 

• City of Mount Vernon Public Schools, Westchester County 

❖ New York State Capitol Region 

• Enlarged City of Troy Public Schools 

In the last year, new sites were opened in 

❖ Fairfax County, Virginia 

❖ Dorchester, Massachusetts 

❖ San Francisco 
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As part of its expansion drive, ISA is in the process of restructuring its 
organization to increase the internal capacity of its national office. Researching an 
organization that is in the process of reorganization and expansion is a complicated 
matter. Even though ISA is going through the turmoil of expanding while renewing its 
self-identify and mission, its leaders and staff members have cooperated in this 
independent evaluation and have embraced the challenge of the thorough self- 
examination that necessarily follows such an evaluation. 



If There Were an “ISA Effect,” How Would We Know It? 

We do not expect to see a long-term “ISA effect” in the here and now. The whole 
premise of ISA is that it is possible to change the trajectory of a young child’s life so that 
child who was, perhaps, prison-bound or destined for unemployment will become a 
contributing member of society. Thus, to really know whether ISA was effective would 
require checking in with the students 30 years later, evaluating the degree of their success 
in life, and seeing whether they attribute that success at least in part to their participation 
in COMET/STAR. 

Despite this, the program can be evaluated now on two overarching issues. First, 
to find out whether ISA meeting its own criteria for success, one can examine whether 
the outcomes that the adults desire for the students are being made more likely by the 
adults’ daily efforts. We can, for example, ask if the adults are promoting the day-to-day 
behaviors that amount to academic perseverance and its close cousin: an orientation 
toward the distant future. Second, to find out whether ISA’s efforts are likely to make a 
difference in the students’ present and future lives, one can compare ISA’s efforts with 
those we already know are most likely to make a difference. For example, the work of 
Dr. James P. Comer and the School Development Program has shown the importance of 
relationships between and among teachers and students, and the importance these 
relationships have for later life success. In assessing the effectiveness of ISA, we 
therefore can ask students if they have high levels of engagement with adults. 

Evaluation Design 

In order to discover if there has been an ISA effect, we used a triangulated approach that 
provided a measure of cross-validation of the findings: 

1. To determine the extent to which positive trends may be discerned with regard to 
student outcomes , we conducted a retroactive analysis on the achievement data of the 
first four ISA cohorts (167 students) and a cohort analysis of the ISA Class of 2000 
from four schools (presented in this paper). 

2. To determine the level and quality of the students’ youth development , we 
administered the Youth Development and Student Learning Inventory ( YDSL ) (1998, 
1999) to 173 ISA students from Roosevelt Union Free School District, Long Island; 
Wyandanch Union Free School District, Long Island; Enlarged City of Troy Public 
Schools, New York State Capitol Region; and City of Mount Vernon Public Schools, 
Westchester County. The responses of the ISA students were compared to responses 
of 698 non-ISA students at a comparison school. 
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3. To determine the quality of program implementation at the sites, we (1) conducted 70 
direct observations of regularly scheduled ISA academic enrichment and counseling 
class periods and (2) administered our Staff Feedback Form to 73 ISA school-based 
staff members. 

Our direct observations of COMET/STAR enrichment and counseling sessions 
and analysis of the assessments of the COMET/STAR faculty regarding their 
effectiveness helped us to interpret the students’ responses to the YDSL. 

We also conducted interviews with ISA staff members at all levels and with 
students throughout the 3 years of the study. 

Data Collection Instruments 

The Youth Development and_ Student Learning Inventory (YDSL) 

The Youth Development and Student Learning Inventory (YDSL) (Ben-Avie and 
Haynes, 1998, 1999), a survey for high school students, is an outgrowth of 10 years of 
research at the Yale Child Study Center on the relationship between youth development 
and student learning. YDSL is a 1 12-item scale with a 5-point Likert response format that 
measures 4 dimensions of student learning and 4 dimensions of youth development. The 
student learning dimensions are academic perseverance, excellence in language, problem 
solving in math and science, and future academic orientation. The youth development 
dimensions are engagement with adults, peer bonding, coping, and professional conduct. 

The internal consistency reliability for the YDSL was found to be in the robust 
high range (.91). The internal consistency reliability for the student learning dimension 
was found to be in the robust high range (.91) and the internal consistency reliability for 
the youth development dimension was found to be in the high range (.89). The 
relationship between the student learning dimension and the youth development 
dimension was observed to be strong (r = .764, p. < .001). 

Staff Feedback Form 

The Staff Feedback Form (Ben-Avie and Steinfeld, 2001) is a 97-item scale with 
a 5-point Likert response format that measures the respondents’ assessment of their 
career, mood in relation to work, work-related experiences of being critiqued and 
observed, ability to maintain appropriate boundaries between job and personal life, 
experience as a mentee and a mentor, personal efficacy, ability to manage anger and 
anxiety, experience with working on a team, and assessment of the workplace from a 
professional perspective. The internal consistency reliability for the SFF was found to be 
in the high range (.86). 

ISA Observation Log 

The ISA Observation Log (Ben-Avie, 1998, 1999) was constructed to document 
the nature and quality of program implementation through direct observations of ISA 
academic enrichment and counseling sessions. Sections include positive learning 
environment, quality of teaching, academic enrichment, and student engagement. 
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Impact Analysis 

Figure 1 shows the model of effects that we developed in order to see whether an ISA 
effect may be discerned. The COMET/STAR team engages in activities that include (1) 
providing students with academic and counseling support services, (2) reaching out to 
parents, and (3) maintaining a positive relationship with the host school. Students who 
actively participate in COMET/STAR demonstrate an increase in academic performance, 
positive behaviors, and self-concept. Increases in academic performance, positive 
behaviors, and self-concept are intervening variables and not outcomes, according to the 
model. As intervening variables, they influence whether or not the students achieve 
desirable immediate and long-term outcomes. For example, graduating from high school 
is an outcome that ISA considers desirable; students’ academic performance influences 
whether or not they will graduate. 

If ISA students receive at least a therapeutic level of treatment, then certain 
outcomes are expected to occur in the here-and-now. Among these immediate outcomes, 
students (1) remain in school and complete middle and high school on time; (2) develop 
competency in higher-order reasoning, thinking, communication, and comprehension 
skills, with an emphasis on reading, writing, math, and science; (3) develop a greater 
sense of self and independence in learning; (4) develop a positive attitude towards school, 
peers, family, and community; and (5) understand how the present course of study affects 
future career options. These immediate outcomes are expected to lead to the following 
long-term (“life-span”) outcomes: Students graduate from high school and either enter 
postsecondary education in good academic standing or initiate a meaningful career path. 
Today we are able to observe intervening and immediate outcomes. It may take years 
before long-term outcomes are discemable. 



Defying Negative Predictions: The Students 

According to ISA’s theory, students who actively participate in COMET/STAR’s support 
services tend to develop a strong connection to the educational process. ISA calls this 
effort and its outcomes “increasing the students’ resilience,” that is, their determination to 
overcome occasional setbacks and stay in school. This increases the likelihood that they 
will complete high school and continue on to postsecondary education, thus increasing 
their options in later life. And helping high school students to complete formal education 
is an urgent need. On February 28, 2001, The New York Times published data in an article 
on high school dropout rates. It read, “the percentage of [New York City] students who 
graduated in four years was down to 49.7% for the class of 2000 from 50.1 for the class 
of 1999” (Hartocollis, p. B6). ISA targets those students who have been identified as 
most likely to drop out of school. In effect, ISA’s mission is to help students defy the 
negative predictions about their future. 

According to the model, students placed at risk of dropping out are referred to 
ISA by the appropriate grade level counselors in June and February, and the school’s 
counselor informs the parents. Who is most at risk of dropping out? A report submitted in 
1997 by the State University of New York and the New York State Department of 
Education, A Report to the Governor and the Legislature on the Educational Status of the 
State’s Schools, cites 4 risk categories: (1) background factors, (2) student performance 
and ability measures, (3) student behaviors and choices, and (4) school factors (p. 153). 
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“Background factors” refers to nontraditional family structure and low socioeconomic 
status. “School performance and ability measures” refers to poor grades and low scores 
on standardized tests. “Student behaviors and choices” refers to patterns of behavioral 
problems in school, teenage marriage and pregnancy, and arrests. “School factors” refers 
to such school characteristics as enrollment, programs, teacher-pupil ratio, percent of 
ethnic minority enrollment, competency testing, and teacher turnover rate. Consistent 
with these criteria, ISA’s eligibility codes include the following: Academic performance, 
attendance, behavior, family/peers, negative family change, abuse/neglect, homelessness, 
substance abuse, limited English proficiency, pregnancy/parenting, negative peer 
pressure. 



1999-2000 Profile 

During the 1 999-2000 school year, the school attendance rate for COMET was 93% and 
the attendance rate for STAR was 94%. Seventy-nine percent of COMET students 
achieved a passing GPA in all 4 core subject subjects combined, as did 71% of STAR 
students (Boykin and Sukhu, 2000). 

Table 1 shows the promotion/retention rates at Roosevelt COMET/STAR, the 
first ISA site, which appear to be typical of ISA. To appreciate the ISA difference, it 
helps to widen the frame for a moment and discuss the state of Roosevelt school district. 
On March 21, 2001, The New York Times reported that the New York State Board of 
Regents recommended a state takeover of Roosevelt school district, calling it “the only 
viable option for raising student achievement and restoring financial stability in the 
chronically failing system” (p. Bl). Up until now, the state has refrained from completely 
taking over a school district. However, the state may do so in the case of Roosevelt 
because the district has failed twice to develop a “convincing” improvement plan (p. B2). 



Table 1. Promotion/Retention Status Report for Roosevelt COMET/STAR (1999-2000) 



Grade 

Level 

1999-2000 


Total 

students 

enrolled 


Number 
promoted in 
June 2000 


Number in 
school July- 
August 2000 


Number 
retained in 
June 2000 


Total promoted 
for school year 
1999-2000 
(computed after 
the summer of 
2000) 


7 


30 


25 


5 


0 


30 


8 


38 


22 


16 


0 


38 


9 


36 


31 


5 


2 


34 


10 


34 


23 


9 


3* 


31 


11 


40 


30 


10 


5 


35 


12 


38 


34 


4 


3 


35 graduates** 


Total 


216 


165 


49 


13 


203 



* 2 students did not attend summer school. 

** 2 out of 3 seniors are scheduled in school as fifth-year students; they did not graduate in four years and 
are returning for another year of high school. 
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Table 2 shows a cohort analysis for ISA’s Class of 2000. Once again, it is critical 
to understand ISA’s graduation rate against the backdrop of the dropout rate in similar 
schools. According to a recent The New York Times article, “most — 67% — of the 
dropouts in the class of 2000 never advanced beyond 9 th or 1 0 lh grade, even though most 
stayed in school for four years” (Hartocollis, March 22, 2001). 



Table 2. Cohort Analysis of the STAR Class of 2000 in Four High Schools (N= 111) 



Site 


Mean overall GPA 
when they were 
Sophomores 
1997-1998 


Mean overall 
GPA when they 
were Juniors 
1998-1999 

1 yi } »?§ 


Mean overall 
GPA when they 
were Seniors 
1999-2000 

! ' wf 

i: - A 


Number of 
graduates 


Hempstead 


67 


66 


68 


27 (87%) 


Roosevelt 


72 


73 


77 


35 (92%) 


Wyandanch 


76 


75 


77 


23 (96%) 


Troy 


72 


74 


76 


18(100%) 


Total 








103 (93%) 



Source of data: Table 13 and Table 14 (Boykin and Sukhu, 2000) 

** Only those sites which had at least 3 data points for the Class of 2000 appear in this table. 



The grade-point averages (GPAs) of the Class of 2000 are similar to those of the 
first 4 ISA cohorts (167 students), which we analyzed to discern whether the positive 
changes that were observed among the first cohorts of ISA may be attributed to chance or 
to the effects of ISA . During the freshman year of the ISA cohorts that were analyzed, the 
mean GPA of the students was 75.14. (The range was from 56 to 94, and the standard 
deviation was 7.49). After three years of high school, the mean GPA of the ISA students 
was 78.14. (The range was from 53 to 94, and the standard deviation was 8.01). A slight 
decrease in mean GPA was observed during the students’ senior year. Given the ISA 
students’ GPAs in their freshman year and the reason why the students were referred to 
COMET/STAR, one might have expected the students to have dropped out of high school 
or experienced school failure. The data analysis suggested an ISA effect because the 
students’ mean GPA over time could not have been predicted using either chance or 
comparison to national, matched groups' (Ben-Avie et al., 2000). 



Can We Attribute the Students’ School Success to an ISA effect? 

A cohort of students who had been identified as likely or most likely to drop out of 
school did not do so. How are we to know whether we may attribute this to an ISA 
effect? The evidence has to be persuasive, taken as a complex, interrelating whole. For 
example, adolescents tend to engage in magical thinking — “Somehow, the project is 
going to get done: Somehow, the college application is going to get completed and sent in 
the mail.” If we see an organization that claims that it is fostering academic perseverance 
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but we observe this kind of magical thinking throughout its student body, then we would 
not accept its claims. Our first step, then, is to take a look at the nature and quality of the 
students’ learning and development, as measured in interviews with students and with 
staff members; by direct observation of academic, counseling, and college and career 
planning meetings; and by administering our own Youth Development and Student 
Learning Inventory. 

Analysis of Responses to the Youth Development 
and Student Learning Inventory (YDSL) 

We designed our research instruments to capture students’ intensity of agreement with 
statements that point to (1) future orientation, (2) academic perseverance, (3) coping, (4) 
engagement with adults, and (5) affiliation with a well-supervised youth group, all of 
which have been demonstrated to promote and/or reflect student learning and youth 
development (see Comer, Ben-Avie et al., 1999). The relationship among these variables 
is complex and interdependent. Knowing what we know about student learning and youth 
development, if we did not see evidence of these variables, we would reject the idea of an 
ISA effect. It would be too implausible that ISA could overcome the students’ prior risk 
factors without program elements that promote high levels of these variables. Therefore, 
these are the variables that we use in the present paper to determine whether we may 
attribute to an ISA effect the students’ defiance of negative predictions that have been 
made about them. 

There are complex links between and among all five of the variables we have 
chosen to evaluate. Consider Future Orientation and Academic Perseverance : People 
learn from many sources what to expect of themselves in the here and now and in the 
future. By the 7 th grade, students placed at risk of failure typically demonstrate that their 
orientation toward the future (Future Orientation) already diverges from the orientation 
of their more-likely-to-be-successful age-mates. It is marked by magical thinking 
(“Somehow, the project will get done in time”), by hoping for an external intermediary 
(“I’ll win the Lottery”), and/or by the expectation that the future will remain largely the 
same as the present (“I’ll always live in a dangerous neighborhood”; “I wouldn’t know 
how to behave in such a fancy office”). All these beliefs can support habits antithetical to 
academic excellence. In order for students to change, and in order for the adults who 
guide them to help them change, the students and adults must have and demonstrate 
thoughts, feelings, and behaviors that support a positive attitude about the long-distant 
future and about themselves as successful learners and contributing members of society. 

During the 1999-2000 school year, we administered the YDSL to 173 ISA 
students from Roosevelt Union Free School District, Long Island; Wyandanch Union 
Free School District, Long Island; Enlarged City of Troy Public Schools, New York State 
Capitol Region; and City of Mount Vernon Public Schools, Westchester County. 

Selected Findings from the YDSL 

ISA’s academic, counseling, and college and career components are designed to 
encourage the development of internal motivation and personal accountability for school 
success. We analyzed ISA students’ responses to the YDSL to see whether a subgroup 
could be discerned based on those who indicated “luck” in response to the following 
item: “Most of the time, when I do well in school the major reason for my success is (1) 
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luck (2) being prepared (3) ability.” Fifty percent responded “ability,” 36% “being 
prepared,” and 14% “luck.” (It is interesting to note that the lower-performing ISA 
students tended to attribute their success in school to luck. Twenty-five percent of the 
ISA respondents indicated that most of the students in their grade have GPAs that are 
“lower” than theirs, 53.8% responded “about the same,” and 21.3% responded “higher.” 
Those ISA students who indicated that most the students in their grade have GPAs that 
are higher than theirs tended to indicate “luck” as the major reason for their success in 
school, significantly differing from those who responded that most other students’ GPAs 
were “about the same as mine” (p. = .02) and those who responded that most other 
students’ GPAs were “lower than mine” (p. = .004).") 

ISA’s academic, counseling, and college and career components are designed to 
encourage students to thinking responsibly and creatively about their future. Eighty-three 
percent of the ISA students who responded to the YDSL strongly agreed or agreed that 
they often think about what they are going to do after high school graduation. Seventy- 
eight percent strongly agreed or agreed that they tend to set goals for their future. 
Seventy-nine percent strongly agreed or agreed that they could imagine themselves in 
college or an institution of higher education. By way of contrast, the variable that had the 
lowest mean score among the non-ISA students at a comparison school was Future 
Orientation. This variable measures the students’ tendency to devote attention to the 
future, set goals beyond the immediate time frame, and engage in behaviors designed to 
reach future goals. 

The responses of ISA students to the items that tap into future orientation are all 
the more remarkable when it is remembered that the inclusion criteria for participation in 
ISA programs stress “most likely to drop out.” Strathman, Boninger, Gleicher, and Baker 
(1994) note the positive correlation found between future orientation and socioeconomic 
status. Students who experience pressing immediate events (e.g., poverty and violence) 
tend not to take action to shape future outcomes. The relationship between potential 
future outcomes and present behavior for students is captured in the following passage by 
Lens and Moreas (1994): 

People with a long future time perspective . . . will experience less 
immediate satisfaction and more delayed satisfaction due to goal 
attainment (e.g., to become a nurse in two years vs. a medical doctor in 
seven years). However, the self-imposed delayed gratification that is 
inherent in long-term goal-setting cannot be reached by waiting. One 
usually has to perform a longer or shorter series of instrumental actions in 
order to achieve one’s goals, (p. 27) 

For high school students, the “instrumental actions” are the daily activities of learning 
that are measured by our Academic Perseverance variable of the YDSL. 

ISA’s academic, counseling, and college and career components are designed to 
support students in their daily activities of learning, the “instrumental actions” mentioned 
above. These are measured by the Academic Perseverance variable of the YDSL. 
Academic Perseverance refers to motivation, persistence, and strategic behaviors that 
increase the likelihood of academic success, regardless of occasional obstacles. A 
significant difference was observed between the students’ score on Academic 
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Perseverance between Site A, which had a mean of 3.2, and the students’ score on the 
same variable at Site B, which had a mean of 3.5 (p = .046). The students’ overall GPA 
echoed this difference. Site A’s 9 th grade students had an overall GPA of 70, in 
comparison with Site B’s 9 th grade students, who had an overall GPA of 75. Similarly, 
Site A’s 10 th grade students had an overall GPA of 69, in contrast to Site B’s 73. 

A comparison of ISA and non-ISA students’ responses to items in the Academic 
Perseverance variable is revealing. 

• While only 39% of non-ISA students strongly agreed or agreed to the 
following item, 55% of ISA students did so: “I double-check my answers 
before I hand in an assignment.” 

• While 66% of ISA students strongly agreed or agreed to the following item, 
59% of non-ISA students did so: “I think through new ideas I leam in class 
until I understand them.” 

• While 41% of non-ISA students strongly agreed or agreed to the following 
item, 34% of ISA students did so: “I make less of an effort in a class where I 
do not like the teacher.” 

ISA counselors intervene, and they teach the students to intervene and sustain 
interventions for themselves. On the YDSL, Coping (which we defined as the ability to 
quickly recover one’s healthiest self during and after challenging social situations) was 
found to be the most important predictor of Academic Perseverance (see Table 3). The 
next two most important predictors of Academic Perseverance were found to be Problem 
Solving in Math and Science and Engagement with Adults. Problem Solving in Math and 
Science is the careful analysis of a problem and the formation of steps toward solving it. 
Engagement with Adults is the tendency to trust adults and to develop supportive 
relationships with them. 



Table 3. Coping was the most important predictor of Academic Perseverance 



Variable 


R 

■ ' : 


R Square 


Adjusted 
R Square 


Beta 


Significance 
of T 


Coping 


.569 


.324 


.318 


.569 


<.0001 


Math/Science 


.674 


.454 


.444 


.389 


<.0001 


Adults 


.693 


.481 


.466 


.190 


.02 



That Coping was observed to be the most important predictor of Academic Perseverance 
is intriguing because ISA directly addresses the students’ coping skills through its 
counseling component. Individual, group, and crisis counseling aims to promote students’ 
personal development, especially coping skills. ISA counselors intervene . A comparison 
of ISA and non-ISA students’ responses to items in the Coping variable is interesting: 

• While 46% of ISA students strongly agreed or agreed to the following item, 
35% of non-ISA students did so: “If I think that a classroom rule is unfair, I 
am able to respectfully get a teacher to consider changing it.” 
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• While 62% of ISA students strongly agreed or agreed to the following item, 
51% of non-ISA students did so: “Many times, when two of my friends have 
argued, they have asked for my help in settling the argument.” 

• While 47% of ISA students strongly disagreed or disagreed with the following 
item, 38% of non-ISA students did so: “When there is a conflict between two 
adults in my family, I am unable to do my schoolwork.” 

COMET/STAR provides students with a sense of belonging. The students feel 
part of a group that has strong adult guidance. A comparison of ISA and non-ISA 
students’ responses to items in the Adult Engagement variable is interesting: 

• While 61% of ISA students strongly agreed or agreed with the following item, 
39% of non-ISA students did so: “When I am having some type of crisis 
during the school day, an adult is there for me.” 

• While 67% of ISA students strongly agreed or agreed with the following 
statement, 42% of non-ISA students did so: “When I am making a decision 
about my future, I get advice from an adult at school.” 



Who Are the Adults? The ISA Team 

To determine whether an effect on student learning may be attributed to ISA, an analysis 
of the nature and quality of the intervention is essential. This analysis begins with a 
description of the adults charged with implementing the intervention. The topic is 
imperative because “Troubled students lack easy access to adults and psychologists.” 
This is the title of a recent USA Today article written by Kevin Dwyer (1999), then 
president-elect of the National Association for School Psychologists. He wrote “Teens 
told me there is no longer an adult at home or in the school to talk to, and that ‘It’s 
getting worse’ ” (p. 15a). ISA students were asked to indicate on the YDSL whether an 
adult is home when they arrive from school. Those students who indicated “never” or 
“rarely” had significantly lower scores on Engagement with Adults (the variable which 
measures students’ tendency to trust adults and to develop supportive relationships with 
them) than those students who indicated “always” or “sometimes” (3.2 vs. 3 .6, p = .03). 
Given that there are students who are not engaged with adults, it is all the more important 
to see who are the adults that ISA brings into the lives of the students. 

Selected Findings of the Staff Survey 

Experience in education. Out of 92 ISA school-based staff members, 73 
completed and returned our Staff Feedback Form (SFF) during January 2001. The 
respondents tended to be experienced educators. Sixty percent of respondents reported 
that they had worked in education for 5 or more years, and less than 10% reported that 
they were new to education. Forty-four percent reported that they had worked in their 
current specialty for 5 years or longer, and only 1 7% reported that they were new to their 
specialty area. 

Future orientation. Most respondents’ self-reports showed them to be good 
models of positive future orientation. Seventy percent of respondents reported that they 
agreed or strongly agreed that “this job is part of a career path that I have planned.” The 




12 

13 



Ben-Avie and Stein feld, Impact of a school-based academic and counseling intervention 



13 



experience of achieving a planned future — and the personal teaching stories this 
achievement generates — are essential for adults who are striving to model for their 
students a future orientation that includes success and satisfaction in the workplace. 

Effects of the job. Most respondents reported that they found the work 
stimulating and were enthusiastic about students, colleagues, and parents. Respondents 
reported positive effects of their work. A large majority agreed or strongly agreed with 
the statements, “My job stimulates my creativity” (82%), “My job stimulates my energy” 
(76%), and “I feel good about myself because of the way I do my job” (almost 87%). 
However, for almost half of the staff members, work took a toll. There was an almost 
even split in response to the statement “My job is physically draining,” with 44% 
agreeing or strongly agreeing and 47% disagreeing or strongly disagreeing. And there 
was a similar split in response to the statement “My job is emotionally draining,” with 
50% agreeing or strongly agreeing and 44% disagreeing or strongly disagreeing. 

Strategies. Two thirds of respondents agreed or strongly agreed with the 
statement “I have a tried-and-true strategy to alter my private thoughts and/or feelings so 
that I actually perform better at work,” and one fifth were not sure. Those respondents 
who disagreed or strongly disagreed with the statement, “I have a tried-and-true strategy 
to alter my private thoughts and/or feelings so that I actually perform better at work” also 
tended: to report that their job was not part of a planned career path ( r = .408 ,p = .001), 
to report that their job did not stimulate their creativity (r = .384, p = .001), to report that 
their job did not stimulate their energy (r = .329, p = .005), to disagree with the 
statement, “I feel good about myself because of the way I do my job” (r = .406 , p = 
.001), to report that they did not welcome supervisors’ observations (r = .255, p = .03), 
and to report that they did not know how to help students create effective strategies (r = 
.287,/? =.01 5). 

Personal boundaries. Many ISA staff members’ responses indicated that they 
felt emotionally safe and were willing to risk being observed and corrected, even by the 
people who evaluated their job performance. Those who agreed or strongly agreed with 
the statement “I welcome being observed by my supervisors” (76%) also tended to: 
welcome being critiqued by supervisors (r = .914, p = .001), feel comfortable being 
observed by peers at work ( r = .524, p = .001), and feel emotionally safe during staff 
meetings (r = .409, p = .00 1 ). 

Relationships. Many ISA staff members’ responses indicated that there is a need 
to improve relationships between colleagues and to make sure that every staff member 
has access to a mentor he or she finds to be helpful. Respondents who disagreed or 
strongly disagreed with the statement “I am comfortable when I critique my peers at 
work” (18%) also tended to disagree or strongly disagree that there was at least one 
person available who could guide them through a problem about a colleague (r = .302, p 
= . 01 ). 

New ideas. Many ISA staff members’ responses indicated that there is a need to 
refresh their optimism and creativity. Those who agreed that they had run out of ideas on 
how to deal with a colleague on site also tended to agree that they: had run out of ideas 
on how to deal with a parent ( r = .303, p = .01), thought that there is a student whom 
nobody could deal with ( r = .356, p = .002), felt that there is a staff member or 
administrator who makes them anxious ( r = .355, p = .002), felt that change will not be 




13 



14 



Ben-Avie and Steinfeld, Impact of a school-based academic and counseling intervention 



14 



forthcoming from administrators on important issues ( r = -.292, p = .01), and felt that 
change will not be forthcoming from administrators on minor issues ( r = -.333, p= .005). 

Expectations. Many staff members’ responses indicated that it is likely that 
negative shifts in their mood are related to their low expectations and, therefore, the way 
they implement the program. Respondents who reported that there was at least one staff 
member or administrator who makes them angry or irritable (46%) tended to: agree or 
strongly agree that they need to learn how to stimulate their creativity when they run out 
of ideas on how to deal with people ( r = .299, p = .013), agree or strongly agree that at 
times they feel that it is simply impossible to fulfill the mission (r = .409, p = .001), 
disagree or strongly disagree that their team does good work together (r = -.306, p = 
.011), and disagree or strongly disagree that their team is making a real difference (r = - 
.433,/?= .001). 

Adult in the workplace. Situated in whatever locations the schools are able to fit 
them, COMET/STAR programs vary in amount and flexibility of space, privacy, and 
separation from the often-disrupting influences of non-ISA students. Large groups of 
respondents reported that they had an acceptable place in which to teach (25%), an 
adequate space in which to work with students (46%), adequate secure space for their 
professional supplies (40%), and adequate private space (51%). Given these responses, 
staff members’ enthusiasm for working with the students (see above) is all the more 
impressive. 

Communications within sites and from the national office to the sites were seen 
by a significant minority to be in need of improvement. There was disagreement or 
strongly disagreement with the statement, “Whenever my supervisor sends around a 
message to staff members, I receive it when I should in order to act on it in time” (27%); 
the ; statement, “ I have easy access to all the documents that I need to do my job well” 
(26%); the statement, “My supervisor has adequately communicated to me the 
performance standards for my job” (17%); and the statement, “The training I have 
received adequately addressed the key duties and tasks of my job” (33%). 

These results show that programs that grow organically, no matter how well- 
conceived and well -implemented at the outset, need institutionalization. ISA staff 
members treasure the freedom they have to be creative in the classroom. However, they 
keenly feel the need for bureaucracy. They desire academic freedom, but on the other 
hand they want the structure: the job policies, the manual, the job hierarchies, the pay 
structure. They want the rules and regulations. To use Margaret Martinez’s (1999) four- 
part description of learning orientations, they are transforming learners as teachers and 
performing , conforming , or resistant learners as adults in the workplace. Transforming 
learners “seldom solely rely on deadlines, structured environments, normative 
performance standards, expected social or instructional compliance, extrinsic rewards, or 
others for learning efficacy or self-motivation.” In contrast to transforming learners, 
“performing learners are short-term, detail, task-oriented learners” and conforming 
learners “learn best in well- structured, directive environments using step-by-step 
procedures.” 

This desire for clear standards is at its root a desire for security and for fairness. 
ISA’s experience shows us that the nuts and bolts of faithful program implementation and 
job performance expectations must be continually updated and re-described at routine 
intervals. All too often, in our focus on the children we neglect these frontline workers. 
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Yet the most successful programs will benefit the adults in the same ways that they 
benefit the students: through clear standards and measurable performance outcomes. The 
school-based staff members need a highly efficient organization surrounding and 
embracing them. There also needs to be a renewal process that keeps the positive energy 
flowing. 



What the Adults Do: The Academic, Counseling, 
and College and Career Support Services 

To document what, specifically, ISA staff members do, we conducted 70 observations of 
COMET and STAR regularly “scheduled periods” (academic enrichment and group 
counseling) during the 1999-2000 school year. Observations were conducted at: 

• Roosevelt Union Free School District, Long Island 

• Hempstead Union Free School District, Long Island 

• Wyandanch Union Free School District, Long Island 

• Community School District Thirteen, Brooklyn 

• City of Mount Vernon Public Schools, Westchester County 

Selected Observations 

Implementation and GPA. We found, for example, that the ISA site that had the 
lowest implementation rating, as measured by our ISA Observation Log, also had the 
lowest overall GPA (68) of all the sites that year. 

Our observations and students’ perceptions. In addition, we noted a close 
correspondence between our direct observations and students’ perceptions, as measured 
by the YDSL. For example, 64% of the students disagreed that when working on a team, 
they often jump in and start speaking before another team member has finished speaking. 
In our direct observations, we noted that in 63% of the sessions this was indeed the case. 
Also, 70% of the students agreed that they ask questions of their teachers in ways that are 
likely to give them the answers that they need. In our direct observations, we noted that in 
74% of the sessions this was indeed the case. 

Tutoring and enrichment. During the 1998-1999 school year, 33% of the ISA 
sessions passed over the threshold from “tutorial” to “enrichment” (tutoring means 
reinforcing the curriculum and enrichment means adding something extra to the 
curriculum). During the 1999-2000 school year, our team categorized 54% of the 
sessions as “enrichment.” 

Quality of teaching. In terms of quality of teaching, we observed that the ISA 
staff members communicated the Class Learning Objective to the students at the 
beginning of the period (85.5% of the sessions), had a clear lesson introduction (86.9%), 
gave the students time to think through a question (88.2%), gave clear directions (91 .2%), 
checked for understanding along the way (89.7%), revised instruction on the basis of 
student comments, questions, and performance (95.8%), provided the students with a 
conceptual framework when presenting a great deal of facts (80.9%), continued the 
lesson until the end of the period (86%), and matched the teaching styles and methods 
with the learning situation of the students (84.3%). 
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Positive learning environment. ISA staff members promoted a positive learning 
environment by creating rapport among the whole group (75.4% of the sessions), 
communicating expectations for achievement (85.5%), ensuring positive interpersonal 
relations and mutual respect (62.9%), and developing an atmosphere that fosters self- 
discipline (63.7%). We also observed that in 91% of the sessions, the students felt 
comfortable letting their staff know when they did not understand. In 73.9% of the 
sessions, the students asked questions of their staff in an appropriate manner, and in 
63.3% of the sessions, the students waited their turn during in-class discussions. In 
general, teamwork was promoted among the students (63.3%). However, during 20 of the 
70 observed ISA sessions, there was an instance of teasing. We also charted the number 
of sessions in which there were inappropriate student behaviors (a category that includes 
teasing) that were not addressed by the staff members, and the number of inappropriate 
behaviors per session. In roughly one third of the 70 observed sessions, multiple 
instances of inappropriate behavior were not addressed by staff members. 

Engagement. Students tended to be engaged during their regularly scheduled 
COMET/STAR period. In 73.5% of the sessions, all the students arrived prepared for 
class. Moreover, the students stayed focused on the lesson until the end of the period 
(70.6%) and the students paid attention when the staff member was speaking (75.4%). 
However, in only 44.7% of the observed ISA sessions did the students keep track of what 
they needed to do, and in only 45.7% of the observed ISA sessions did all the students 
remain engaged. 

Staff commitment. We observed the “STAR Patrol” in action. STAR Patrol is 
one student’s description of the Hempstead STAR staffs tenacity in making sure that the 
students fulfill their responsibilities. Staff members support program goals most 
effectively by making sure that the students actually attend class and after-school 
programs for which they are scheduled and by following up with the family when 
students cut class, are absent from school, do not complete homework, or have other 
difficulties. 

Youth group. We observed that COMET/STAR functions for the students as a 
youth group or alternative peer group. Some notable features of youth groups include a 
staff that knows each student well and a sense of belonging to a group. This becomes 
especially clear when we observe that ISA students tend to eat lunch together in the 
cafeteria and walk down the halls together (Ben-Avie, et al., 2000). “Knowing students 
well” means that the staff members make the effort to know about the students’ out-of- 
school lives. 111 One especially effective way in which COMET/STAR engenders this 
youth group function is through day trips and overnight trips. These trips contribute to the 
students’ enrichment as much as (if not more than) the regularly scheduled 
COMET/STAR periods. As we noted, “ISA’s answer to the question of how to help 
students placed at risk to successfully complete formal education is theoretically sound: 
Promote students’ participation and belonging in a youth group that has clearly defined 
expectations of behavior while providing them with the tools to achieve academically” 
(Ben-Avie, et al., 1999, p. 42). 

Team support for future goals. We observed that during their junior and senior 
years of high school, ISA students participate in a well-formulated college and career 
preparation curriculum. This curriculum includes weekly meetings designed to raise the 
educational aspirations of the students, internships, SAT workshops, college tours, self- 
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appraisal sessions, and a guided process of applying to college or initiating a meaningful 
career path. To increase the probability that those students who were identified as likely 
to drop out of school during their first or second year of high school succeed in reaching 
the college and career component, ISA provides intensive team support. It appears that 
the underlying reason for an effect of the ISA College and Career component is that this 
component stretches the students’ future orientation. 



Can We, Then, Attribute the Findings to an ISA Effect? 

We set out to determine whether ISA programs work, and, if so, why do they work. Our 
answer is as follows: When there is an affiliation with a well-conceived and well-led 
group, adults and students will continue, up to a point, to come back to a very 
difficult situation with the expectation of success. The ISA effect is clearly 
observable on a variety of different scales. Even with its enthusiastic, unplanned 
growth, ISA is a model for certain program elements without which students at risk 
cannot be expected to succeed in school and, therefore, in their later lives. At its 
best, when implemented most faithfully, ISA has fulfilled its mission. 



Implications 

It is clear from our observations that students who have been identified as being “at risk 
of not making healthy transitions” do not stop being “at risk” simply because they 
participate in an intervention. At what point do “at-risk” kids stop being at risk? Is it after 
the student transitions to a more stable environment? Is it after the student has become 
“other” than what he or she was before (e.g., more self-motivated, more connected to 
school)? Or, is it when the student transitions to adulthood? We asked James P. Comer, 
M.D., founder of the Yale School Development Program, to respond to this question of 
at-riskness. He told us that children transition to a state of no longer being at risk when 
they are able to manage the average expected environment. All of us live in a world that 
presents a series of challenges. Most of us are prepared to handle these challenges. A 
child who is not prepared to handle challenges is at risk. Therefore, a twofold 
intervention is needed that simultaneously prepares children and reduces risks inherent in 
the environment (e.g., reduces violence in the school). Our evaluation of the Institute for 
Student Achievement’s programs in junior and senior high schools has yielded these 
insights: 

A disruptive school environment will undermine any intervention program, 
no matter how well-conceived and well-implemented. While ISA students routinely 
describe the COMET/STAR room in their school as an “oasis,” it is impossible (and, 
perhaps, not desirable) for ISA to try and close itself off from the rest of the school. What 
we observe in ISA is that, at all levels, the efforts of people implementing the program 
are so frequently impeded by interruptions, violence, illness — the daily life of adults and 
students in low-performing schools — that they are often amazed when their students are 
able to succeed. Thus, the response has to be twofold: (1) The team has to include a 
sufficient number of persons (in ISA’s case, social workers and other similarly trained 
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professionals) who understand child and youth development so that barriers to academic 
success can be removed; and (2) there needs to be a parallel track for the school (e.g., 
staff training in teaching methods and youth development; access to support groups). 

There is a threshold beyond which unplanned growth undermines program 
implementation. The Institute for Student Achievement began as a drop-out prevention 
program at one high school. During the first decade of its growth, ISA expanded from 
New York to sites in California, Virginia, and Massachusetts. ISA has completed its 
pilot, early replication, and initial expansion stages, and is now poised to emerge as a 
national presence — or to collapse under its own weight. What ISA brings to the table is 
(1) the group’s experience and expertise in working with the “hardest to reach” students 
and (2) its experience in acting as a broker between corporate American and public 
education. ISA needs to document its experience and effectiveness in these arenas as 
much as it documents success on the individual student level. 

Meaningful educational change results in potentially life-long change in the 
individual student. Academic goal-setting is linked to locus of control — and locus of 
control is linked to future orientation. “Individuals who engage in self-regulated 
academic goal-setting are more likely to judge that their outcomes are caused by their 
own efforts (internal attributions) or factors outside of their agency (external 
attributions),” write Terell P. Lasane and James M. Jones (1999). Citing previous 
research, Lasane and Jones add that “future time oriented students were more likely to 
make internal attributions for success (effort and ability) than external attributions (luck, 
task characteristics)” (p. 33). ISA has shown that high school students who have been 
.. identified as “most likely to drop out” can transform themselves “so that they can succeed 
in our society” (to quote ISA’s mission) when they develop in the here-and-now high 
levels of future orientation, coping with appropriate tasks, and academic perseverance, as 
well as skills for coping with failure and disappointment without giving up — that 
students can learn how to see beyond the next step, and to discover what steps they need 
to take in the present in order to accomplish their most-desired future. In other words, 
when implemented at its best, ISA has demonstrated that students who previously were 
either hopeless or unaware of what to hope for can move into their futures with optimism, 
options, and skills. 

Sustained commitment. The major policy implication of ISA is clear: Programs 
must sustain their commitment to children and youth over many years. At a time in which 
our educational system is fragmented, ISA walks with the students from middle school 
through high school graduation. ISA teaches us that if you want to make a difference in 
the lives of children, you can’t jump in and out of their lives. You also can’t expect major 
changes in the trajectories of children’s lives after only a year or so of intervention. 
Meaningful change — true change — takes well-implemented commitment over time. 
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Endnotes 

1 According to Achievement in America 1999 , data provided to us by The Education Trust in 

Washington, D.C.: Of every 100 White Kindergartners: 93 graduate from high school, 62 complete at least 
some college, and 29 obtain at least a bachelor’s degree. Of every 100 African-American kindergartners: 86 
graduate from high school, 48 complete at least some college, and 15 obtain at least a bachelor’s degree. Of 
every 100 Latino kindergartners: 61 graduate from high school, 31 complete at least some college, and 10 
obtain at least a bachelor’s degree. (These figures are based on current 24 year olds). Of those students in 
the lowest test quartile, only 27% enter 2-year colleges and only 12% enter 4-year colleges, in contrast to 
the students in the highest quartile of which 12% enter 2-year colleges and 77% enter 4-year colleges. 
Fifty-five percent of African-American students and 58% of Latino students enroll in college directly after 
high school graduation, in contrast to 67% of white students. 

The National Education Longitudinal Study 1988-1994, Descriptive Summary Report , describes a 
study conducted by the National Center for Education Statistics (NCES) of students who were eighth 
graders in 1988 and who were deemed at-risk of dropping out of high school due to socioeconomic and 
family circumstances that existed at the time. According to the report, the factors constituting risk were 
whether the student: 

• lived in a single-parent family 

• was from a family with an annual income of less that $ 1 5,000 

• had an older sibling who had dropped out of school 

• had parents who did not finish high school 

• had limited proficiency in English and/or 

• was at home without adult supervision more than three hours a day 

The NCES notes that these five selected factors are only some of the many factors that could be used to 
determine which students are “at-risk.” The factors used in the report were the same ones used in a number 
of previous NCES reports. 1 



Moreover, 

• Those who indicated “luck” as the major reason for their success had a mean score of 3. 1 on Academic 
Perseverance , significantly lower than those students who indicated “being prepared” (3.4). 

• Those who indicated “luck” had a mean score of 3. 1 on Engagement with Adults , significantly lower 
than those students who indicated “ability” (3.7). 

• Those who indicated “luck” had a means score of 3.0 on Coping , significantly lower than those 
students who indicated “ability” (3.3) and “being prepared” (3.3). 

• Those who indicated “luck” had a mean score of 3.2 on Professional Conduct , significantly lower than 
those students who indicated “ability” (3.7) and “being prepared” (3.7). 

• Those who indicated “luck” had a mean score of 3.0 on the overall Youth Development dimension, 
significantly differing from those who indicated “ability” (3.4) and “being prepared” (3.4). 



And this is important because we found in our analyses of ISA students’ responses to the Youth 

Development and Student Learning Inventory that: 

• Those students who indicated that they do not have a part-time job had a significantly higher mean 
score on the overall Youth Development dimension (3.4) than those students who work 6-10 hours each 
week (2.9). This does not mean that working is not good for youth development: the students who had 
the highest mean score were those who worked between 1 through 5 hours each week. 

• Those students who indicated that they worked 6-10 hours each week also had the lowest mean score 
on the overall Student Learning dimension. The students who had the highest mean scores were those 
who did not have a part-time job or worked between 1 to 5 hours each week. 
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• Those students who indicated that they usually take care of a younger brother or sister or other relative 
had significantly higher scores on overall Youth Development (p. = .002) and overall Student Learning 
(p. = <.00 1) and the overall YDSL (p. = .002). 

• Those students who indicted that they are routinely responsible for helping an older relative (a person 
who is not their parent or guardian) had significantly higher scores on overall Youth Development (p. = 
.007). 
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